PS-11020 [8.0]: MyRocks - concurrent ADD INDEX could pollute L0 or crash the server.#5921
PS-11020 [8.0]: MyRocks - concurrent ADD INDEX could pollute L0 or crash the server.#5921polchawa-percona wants to merge 5 commits into
Conversation
… the server. Problem: Concurrent ADD INDEX could lead to L0 level being polluted with the bulk-loaded data that was supposed to go to LMAX. Moreover, if the user has set rocksdb_bulk_load_fail_if_not_bottommost_level session variable to true, the server could crash when concurrently running ADD INDEX. The same issues occurred when ADD INDEX was running while a new table was concurrently created. Solution: Serialize bulk loads together with all index id allocations so they could ingest SST files only having guarantee that all RocksDB keys are the highest when inserted (to avoid risk of collision with overlapping key ranges in merged SST files that include both smaller and higher index id values). Wait when ingesting bulk loaded SST, or when creating a table up, until all bulk loads for smaller index id are finished.
|
This is draft of PR - I still want to double check that I don’t need to remove index id in other cases than finish_bulk_load and I want to add some assertions. |
polchawa-percona
changed the title
|
Commits reviewed:
Scope:
Findings1. High: multi-index
|
- always reclaim index id reservation in finish_bulk_load - finish_bulk_load is invoked within a for-loop (not outside)
2 participants
Problem:
Concurrent ADD INDEX could lead to L0 level being polluted with the bulk-loaded data that was supposed to go to LMAX. Moreover, if the user has set rocksdb_bulk_load_fail_if_not_bottommost_level session variable to true, the server could crash when concurrently running ADD INDEX.
The same issues occurred when ADD INDEX was running while a new table was concurrently created.
Solution:
Serialize bulk loads together with all index id allocations so they could ingest SST files only having a guarantee that all RocksDB keys are the highest when inserted (to avoid the risk of collision with overlapping key ranges in merged SST files that include both smaller and higher index id values).
Wait when ingesting bulk-loaded SST, or when creating a table, until all bulk loads for the smaller index id are finished.